<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Face swapping using face landmark detection</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d3/d81/tutorial_contrib_root.html">Tutorials for contrib modules</a></li><li class="navelem"><a class="el" href="../../de/d27/tutorial_table_of_content_face.html">Tutorials for face module</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Face swapping using face landmark detection </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>This application lets you swap a face in one image with another face in other image. The application first detects faces in both images and finds its landmarks. Then it swaps the face in first image with in another image. You just have to give paths to the images run the application to swap the two faces. </p><div class="fragment"><div class="line">// Command to be typed for running the sample</div><div class="line">./sample_face_swapping -file=trained_model.dat -face_cascade=lbpcascadefrontalface.xml -image1=/path_to_image/image1.jpg -image2=/path_to_image/image2.jpg</div></div><!-- fragment --> <h3>Description of command parameters</h3>
<blockquote class="doxtable">
<ul>
<li><b>image1</b> i1 (REQUIRED) Path to the first image file in which you want to apply swapping.</li>
<li><b>image2</b> i2 (REQUIRED) Path to the second image file in which you want to apply face swapping.</li>
<li><b>model</b> m (REQUIRED) Path to the file containing model to be loaded for face landmark detection.</li>
<li><b>face_cascade</b> f (REQUIRED) Path to the face cascade xml file which you want to use as a face detector. </li>
</ul>
</blockquote>
<h3>Understanding the code</h3>
<p>This tutorial will explain the sample code for face swapping using OpenCV. Jumping directly to the code :</p>
<div class="fragment"><div class="line">CascadeClassifier face_cascade;</div><div class="line">bool myDetector( InputArray image, OutputArray ROIs );</div><div class="line"></div><div class="line">bool myDetector( InputArray image, OutputArray ROIs ){</div><div class="line">    Mat gray;</div><div class="line">    std::vector&lt;Rect&gt; faces;</div><div class="line">    if(image.channels()&gt;1){</div><div class="line">        cvtColor(image.getMat(),gray,COLOR_BGR2GRAY);</div><div class="line">    }</div><div class="line">    else{</div><div class="line">        gray = image.getMat().clone();</div><div class="line">    }</div><div class="line">    equalizeHist( gray, gray );</div><div class="line">    face_cascade.detectMultiScale( gray, faces, 1.1, 3,0, Size(30, 30) );</div><div class="line">    Mat(faces).copyTo(ROIs);</div><div class="line">    return true;</div><div class="line">}</div></div><!-- fragment --><p> The facemark API provides the functionality to the user to use their own face detector to be used in face landmark detection.The above code creartes a sample face detector. The above function would be passed to a function pointer in the facemark API.</p>
<div class="fragment"><div class="line">Mat img = imread(image);</div><div class="line">face_cascade.load(cascade_name);</div><div class="line">FacemarkKazemi::Params params;</div><div class="line">params.configfile = configfile_name;</div><div class="line">Ptr&lt;Facemark&gt; facemark = FacemarkKazemi::create(params);</div><div class="line">facemark-&gt;setFaceDetector(myDetector);</div></div><!-- fragment --><p> The above code creates a pointer of the face landmark detection class. The face detector created above has to be passed as function pointer to the facemark pointer created for detecting faces while training the model. </p><div class="fragment"><div class="line">vector&lt;Rect&gt; faces1,faces2;</div><div class="line">vector&lt; vector&lt;Point2f&gt; &gt; shape1,shape2;</div><div class="line">float ratio1 = (float)img1.cols/(float)img1.rows;</div><div class="line">float ratio2 = (float)img2.cols/(float)img2.rows;</div><div class="line">resize(img1,img1,Size(640*ratio1,640*ratio1),0,0,INTER_LINEAR_EXACT);</div><div class="line">resize(img2,img2,Size(640*ratio2,640*ratio2),0,0,INTER_LINEAR_EXACT);</div><div class="line">Mat img1Warped = img2.clone();</div><div class="line">facemark-&gt;getFaces(img1,faces1);</div><div class="line">facemark-&gt;getFaces(img2,faces2);</div><div class="line">facemark-&gt;fit(img1,faces1,shape1);</div><div class="line">facemark-&gt;fit(img2,faces2,shape2);</div></div><!-- fragment --><p>The above code creates vectors to store the detected faces and a vector of vector to store shapes for each face detected in both the images.It then detects landmarks of each face detected in both the images.the images are resized as it is easier to process small images. The images are resized according their actual ratio.</p>
<div class="fragment"><div class="line">vector&lt;Point2f&gt; boundary_image1;</div><div class="line">vector&lt;Point2f&gt; boundary_image2;</div><div class="line">vector&lt;int&gt; index;</div><div class="line">convexHull(Mat(points2),index, false, false);</div><div class="line">for(size_t i = 0; i &lt; index.size(); i++)</div><div class="line">{</div><div class="line">    boundary_image1.push_back(points1[index[i]]);</div><div class="line">    boundary_image2.push_back(points2[index[i]]);</div><div class="line">}</div></div><!-- fragment --><p>The above code then finds convex hull to find the boundary points of the face in the image which has to be swapped.</p>
<div class="fragment"><div class="line">vector&lt; vector&lt;int&gt; &gt; triangles;</div><div class="line">Rect rect(0, 0, img1Warped.cols, img1Warped.rows);</div><div class="line">divideIntoTriangles(rect, boundary_image2, triangles);</div><div class="line">for(size_t i = 0; i &lt; triangles.size(); i++)</div><div class="line">{</div><div class="line">    vector&lt;Point2f&gt; triangle1, triangle2;</div><div class="line">    for(int j = 0; j &lt; 3; j++)</div><div class="line">    {</div><div class="line">        triangle1.push_back(boundary_image1[triangles[i][j]]);</div><div class="line">        triangle2.push_back(boundary_image2[triangles[i][j]]);</div><div class="line">    }</div><div class="line">    warpTriangle(img1, img1Warped, triangle1, triangle2);</div><div class="line">}</div></div><!-- fragment --><p>Now as we need to warp one face over the other and we need to find affine transform. Now as the function in OpenCV to find affine transform requires three set of points to calculate the affine matrix. Also we just need to warp the face instead of the surrounding regions. Hence we divide the face into triangles so that each triiangle can be easily warped onto the other image.</p>
<p>The function divideIntoTriangles divides the detected faces into triangles. The function warpTriangle then warps each triangle of one image to other image to swap the faces.</p>
<div class="fragment"><div class="line">vector&lt;Point&gt; hull;</div><div class="line">for(size_t i = 0; i &lt; boundary_image2.size(); i++)</div><div class="line">{</div><div class="line">    Point pt((int)boundary_image2[i].x,(int)boundary_image2[i].y);</div><div class="line">    hull.push_back(pt);</div><div class="line">}</div><div class="line">Mat mask = Mat::zeros(img2.rows, img2.cols, img2.depth());</div><div class="line">fillConvexPoly(mask,&amp;hull[0],(int)hull.size(), Scalar(255,255,255));</div><div class="line">Rect r = boundingRect(boundary_image2);</div><div class="line">Point center = (r.tl() + r.br()) / 2;</div><div class="line">Mat output;</div><div class="line">img1Warped.convertTo(img1Warped, CV_8UC3);</div><div class="line">seamlessClone(img1Warped,img2, mask, center, output, NORMAL_CLONE);</div><div class="line">imshow(&quot;Face_Swapped&quot;, output);</div></div><!-- fragment --><p>Even after warping the results somehow look unnatural. Hence to improve the results we apply seamless cloning to get the desired results as required.</p>
<h3>Results</h3>
<p>Consider two images to be used for face swapping as follows :</p>
<h2>First image </h2>
<div class="image">
<img src="../../227943776_1.jpg" alt="227943776_1.jpg"/>
</div>
<h2>Second image </h2>
<div class="image">
<img src="../../230501201_1.jpg" alt="230501201_1.jpg"/>
</div>
<h2>Results after swapping </h2>
<div class="image">
<img src="../../face_swapped.jpg" alt="face_swapped.jpg"/>
</div>
 </div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:38 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
